The Principles of First Order Automatic Differentiation

This article provides a short overview of the theory of First Order Automatic Differentiation (AD) for readers unfamiliar with this topic. In particular, we summarize different characterisations of Forward AD, like the vector-matrix based approach, the idea of lifting functions to the algebra of dual numbers, the method of Taylor series expansion on dual numbers and the application of the push-forward operator. We give short, but precise mathematical descriptions of these methods and show why they all reduce to the same actual chain of computations (and are, hence, equivalent). Finally, we give a short summary of Reverse AD and again point out the underlying computational steps.